Overview
Code coverage describes the degree to which the source code of a program has been tested.
The basic information about code coverage can be found at http://www.bullseye.com/coverage.html .
Code coverage runs in most frameworks in these phases :
instrumentation changes the bytecode of a classes and inject custom calls to the program
run of tests and collection of metrics reported by the injected custom calls
processing of the metrics
reporting of results
The main java projects which are working or worked on code coverage are :
JVMDI, JBLANKET, InsectJ, noUnit, Hansel, gretel, Groboutils, Quilt, Codecover, Coverclipse, Jester, Cobertura, Emma, Clover
The criteria for selection of right tool for our projects are:
Maven integration
Hudson integration
Reporting in HTML
Fast processing
Good documentation
Progress on project support and development
The mentioned tools will be divided into there groups :
a) Tools that don't fullfill criteria
dead tools
-JVMDI, JBLANKET, InsectJ, noUnit, Hansel, gretel, Groboutils, quilt
different testing method
-jester
other
-codecover, coverclipse
licensed
Jcoverage
Clover
b) Tools that fullfils criteria
Clover
Cobertura
Emma
Dead tools
QUILT
Source |
|
Description |
Quilt is a Java software development tool that measures coverage , the extent to which unit testing exercises the software under test. |
Start |
Ant, Maven |
Technology |
statement coverage |
Frameworks |
BCEL |
Reports |
|
Report example |
|
Last release |
20 October 2003 |
License |
Apache License, Artistic license |
IDE support |
No |
GROBOUTILS
Source |
|
Description |
The popular tools included with GroboUtils include multi-threaded tests, hierarchial unit tests, and a code coverage tool. |
Start |
Ant |
Technology |
logging statements for tracking coverage |
Frameworks |
BCEL |
Reports |
XML, xsl-t transformations, HTML |
Reports example |
http://groboutils.sourceforge.net/codecoverage/CoverageReport-LineCount.html |
Last release |
2004-11-05 |
Licence |
MIT licence |
Additional information |
|
GRETEL
Source |
http://www.cs.uoregon.edu/research/perpetual/dasada/Software/Gretel/ |
Description |
Gretel is a test coverage monitoring tool for Java programs. The current version provides statement |
Start |
GUI, command line |
Technology |
|
Frameworks |
BCEL |
Reports |
GUI |
Last realease |
June 2002 |
License |
BSD |
HANSEL
Source |
|
Description |
Hansel is an extension to JUnit that adds code coverage testing to the testing framework. |
Start |
ANT |
Technology |
|
Frameworks |
ASM |
Reports |
|
Last release |
2006-10-04 |
License |
BSD |
Aditional Information |
|
NOUNIT
Source |
|
Description |
NoUnit allows you to see how good your JUnit tests are. It generates a report from your code to graphically show you how many of your project's methods are being tested , and how well |
Start |
command line |
Reports |
XML, xsl, HTML |
Report example |
|
Last release |
2003-12-15 |
License |
GNU GPL |
IDE support |
|
Addition information |
dont need source codes |
Different technology
JESTER
Source |
|
Description |
Jester doesn't watch which lines of code have been executed. |
Start |
|
Technology |
mutation testing |
Frameworks |
|
Reports |
easy HTML |
Report example |
|
Last release |
jester 2005-04-02, simple-jester 2009-11-24 |
License |
open license (http://jester.sourceforge.net/license.html) |
IDE support |
http://www.dcs.shef.ac.uk/intranet/teaching/projects/archive/ug2005/abs/u2sl.htm |
Additional information |
|
Other
COVERLIPSE
Source |
|
Description |
Coverlipse is an Eclipse plugin that visualizes the code coverage of JUnit Tests. |
Start |
Eclipse |
Technology |
Statement Coverage |
Frameworks |
|
Reports |
Eclipse, java editor |
Reports example |
|
License |
CPL |
Last release |
02-14-2009 |
Comments |
only eclipse support |
CODECOVER
Source |
|
Description |
|
Start |
Eclipse, Command line, ant |
Technology |
CodeCover measures statement, branch, loop, term coverage (subsumes MC/DC), question mark operator coverage, and synchronized coverage |
Frameworks |
|
Reports |
template engine Velocity, html, csv |
Report example |
http://codecover.org/documentation/tutorials/SimpleJavaAppReport.html |
Last release |
19.04.2010 |
Licence |
Eclipse Public Licence (EPL) |
Comments |
- there is no maven support |
Licensed
JCoverage
Clover
CLOVER
Source |
|
Description |
Clover delivers actionable Java code coverage metrics to assess the impact of your tests. It goes beyond raw test coverage to help you focus your limited testing resources where they will be most effective. |
Start |
Eclipse, Netbeans, IntelliJ Command line, ant, maven |
Technology |
Instrumentation of source codes |
Frameworks |
|
Reports |
html, xml, pdf, json |
Report example |
|
Last release |
9-September-2009 |
Licence |
Commercial, sale for acamedics, free for non-profit and open source projects |
Comments |
- Most complete reports and statistics |
COBERTURA
Source |
|
Description |
Cobertura is a free Java tool that calculates the percentage of code accessed by tests. |
Start |
Ant,Command line, Maven |
Technology |
McCabe cyclomatic code complexity of each class (http://en.wikipedia.org/wiki/Cyclomatic_complexity) |
Frameworks |
ASM |
Reports |
Can generate reports in HTML or XML. |
Report example |
|
Last release |
last version 1.9.4.1 released 2010-03-03 |
License |
GNU GPL 2 |
IDE support |
http://ecobertura.johoop.de/ |
Additional Information |
|
Comment |
|
EMMA
Source |
|
Description |
EMMA is an open-source toolkit for measuring and reporting Java code coverage. EMMA distinguishes itself from other |
Start |
ANT , EMMA maven plugin ( The maven EMMA plugin simply wraps the EMMA Ant tasks, so it behaves in a very similar way), |
Technology |
|
Reports |
plain text, HTML (source code linking), XML |
Report example |
http://emma.sourceforge.net/coverage_sample_a/index.html http://emma.sourceforge.net/coverage_sample_a/index.html |
Support |
No write transaction on CVS during last year |
License |
CPL |
IDE support |
support for eclipse EclEmma (http://sourceforge.net/projects/eclemma/ http://sourceforge.net/projects/eclemma/ ) |
Comments |
|
From mentioned frameworks Clover provides the best services. Clover has the best reports, good documentation and easy to use configuration.
As a open source project we can use the product for free. To get the free license the project must be registred and approved by atlassian.
There is also term in license that code coverage reports must be publicly accessible.
Emma provides good enough functionality but the last release was at 2005-07-27 and there is no progress on the project anymore.
Cobertura provides all functions which are needed for code coverage. The main difference between Clover and Cobertura is in reports.
Cobertura report task provides the same basic data but clover offers much more statistics about projects and coverage.
Cobertura maven plugin has also problems with coverage of integration tests, but there is a workaround for this problem.
I vote for Cobertura because it is open source project and provides sufficient data about code coverage of our project.
(Interesting tool is also Jester which uses mutation testing, but it is not what we are looking for right now.)